Detecting apparatus, detecting system, and detecting method

ABSTRACT

The present invention provides a detecting apparatus for detecting an abnormal state such as a drop or fall of a person to be observed from a captured image in a real-time manner and realizing improvement in precision of the detection while eliminating the influence of a background image and noise. The detecting apparatus calculates a motion vector of each of blocks of an image of video data, and extracts a block group in which the size of the motion vector exceeds a predetermined value. The detecting apparatus forms a group from adjacent blocks. For example, in descending order of the area of the blocks, the detecting apparatus calculates characteristic amounts such as an average vector, a variance, and a rotation direction component of motion blocks included in the blocks. The detecting apparatus detects that the person to be observed is in an abnormal state such as a drop or fall on the basis of the characteristic amounts of the groups, and informs an external apparatus or the like of the detection result. The detecting apparatus corrects a deviation of the angle of the imaging direction on the basis of a process of reducing pixels in the horizontal direction of an image and acceleration of the camera, thereby improving the precision of the detection.

CROSS-REFERENCE TO RELATED APPLICATIONS

The disclosure of Japanese Patent Application No. 2013-239115 filed onNov. 19, 2013 including the specification, drawings and abstract isincorporated herein by reference in its entirety.

BACKGROUND

The present invention relates to a detecting apparatus, a detectingsystem, and a detecting method for detecting a state such as of a personto be observed and, more particularly, to a technique of automaticallydetecting the state of a person to be observed by using an image.

There are cases such that an unexpected accident occurs in a space in ahouse or the like where a resident lives and the resident is injured ordead by the accident. It is known that the ratio of a drop and a fall isrelatively high as the causes of those accidents. Particularly, in thecase of elderly people, a drop or fall occurs also in a relatively flatplace such as a living room or a hospital room. In the case of elderlypeople, there are many cases that an accident such as a drop or fallleads to death or a serious disorder. It is consequently important toefficiently find an abnormal situation such as a drop and promptlyhandle it in order to improve the living environment of elderly people.If the state of the person to be observed can be known anytime, anobserver who wants to check the state of a person to be observed such asan elderly person can immediately deal with the abnormal state of theperson to be observed.

Various systems for detecting an abnormal state of a person to beobserved are examined. It is desirable to provide a detecting systemwhich has an affinity to daily life of a person to be observed. Forexample, a system which gives consideration also to the privacy of aperson to be observed while avoiding a physical burden such asattachment of a measuring device to the person to be observed as much aspossible may be provided.

As a technique for promptly finding a drop or the like of an elderlyperson or the like, for example, there is Japanese Unexamined PatentApplication Publication No. 2002-232870 (patent literature 1). Accordingto a technique described in the patent literature 1, a detectingapparatus captures images of a person to be observed every predeterminedinterval and records them. The detecting apparatus calculates a motionvector for each of pixels on the basis of image data. The detectingapparatus calculates the sum of components in the gravity forcedirection of the motion vectors calculated for the pixels. The detectingapparatus preliminarily stores the size of a change of a motion vectorin a predetermined period in the case where a person to be observeddrops. The detecting apparatus compares the sum of the components in thegravity force direction of the motion vectors calculated with thethreshold and, in the case where the component in the gravity forcedirection of the motion vector is larger than a threshold T, determinesthat the person to be observed drops. In the case where it is determinedthat the person to be observed drops, the detecting apparatus performs aprocess for notifying of the drop of the person to be observed. In sucha manner, according to the technique described in the patent literature1, a drop or the like of an elderly person or the like can be promptlyfound.

PRIOR ART LITERATURE Patent Literature

-   Patent Literature 1: Japanese Unexamined Patent Application    Publication No. 2002-232870

SUMMARY

According to the technique described in the patent literature 1,however, the sum of the components in the gravity force direction of themotion vectors using the motion vectors of an entire captured image asobjects and the threshold are compared to determine whether a drop ofthe person to be observed such as an elderly person occurs or not.Consequently, according to the technique described in the patentliterature 1, due to a motion of a background image, camera noise, orthe like, a motion vector in the gravity force direction is generatedwith no relation to the state of the person to be observed, and there isa case that a drop of the person to be observed is erroneously detected.Therefore, a technique which increases the precision of detection whiledetecting an abnormal state of a person to be observed from a capturedimage in a real-time manner is needed.

The other problem and a novel feature will become apparent from thedescription of the specification and appended drawings.

A detecting apparatus according to an embodiment is to detect a motionstate of a person to be observed from an image captured. The detectingapparatus includes an input/output unit, a memory, and a control unit.The detecting apparatus receives an input of video data by theinput/output unit, and stores it into the memory. The control unitcontrols a process of detecting a motion state of the person to beobserved on the basis of the video data stored in the memory. Thecontrol unit calculates a motion vector of each of a plurality of blocksof an image including the video data. The control unit extracts a blockgroup as a set of the blocks in each of which the size of the motionvector exceeds a predetermined value on the basis of the calculatedmotion vectors. The control unit detects an abnormal motion state of theperson to be observed on the basis of the motion vectors of the blocksincluded in the extracted block group. In the case where the abnormalmotion state of the person to be observed is detected, the control unitmakes an informing unit output a signal indicative of an abnormal stateof the person to be observed.

The detecting apparatus according to the embodiment can detect anabnormal state such as a drop of the person to be observed in areal-time manner and notify an observer of the abnormal state whileincreasing the precision of the detection.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of a detectingsystem.

FIG. 2 is a block diagram illustrating the detailed configuration of anMCU (Micro Control Unit) 17.

FIG. 3 is a block diagram illustrating functions of a detectingapparatus 200.

FIGS. 4A to 4F are diagrams illustrating outline of the functions of thedetecting apparatus 200.

FIG. 5 is a diagram illustrating the data structure of groupinginformation 51 and the data structure of characteristic amountinformation.

FIG. 6 is a flowchart illustrating processes for detecting a drop orfall of a person 500 to be observed on the basis of an image captured bya camera 11 by a control unit 22 of the detecting apparatus 200.

FIG. 7 is a flowchart illustrating a thinning process of thinning pixelsof each of images including video data 41.

FIGS. 8A to 8D are diagrams illustrating the directions of scans in thethinning process.

FIGS. 9A to 9C are diagrams illustrating outline of a process ofcalculating motion vectors of each of blocks of an image.

FIG. 10 is a flowchart illustrating processes of calculating an averagevector of motion vectors of blocks included in a group.

FIG. 11 is a flowchart illustrating processes of calculating, as acharacteristic amount, variance of motion vectors, on the basis of themotion vectors of blocks included in a group.

FIG. 12 is a flowchart illustrating processes of calculating a value ofa component of a rotation direction made by a motion vector of eachblock on the basis of the motion vectors of blocks included in thegroup.

FIGS. 13A to 13C are diagrams illustrating an example of distribution ofthe motion vectors of blocks included in a group.

FIG. 14 is a flowchart illustrating processes of detecting the state ofan abnormal motion of the person 500 to be observed by the detectingsystem 100.

FIGS. 15A to 15F are diagrams illustrating an example of the state of adrop or fall of a person to be observed.

FIG. 16 is a flowchart illustrating processes of a drop or fall of theperson 500 to be observed on the basis of an image captured by thecamera 11 by a detecting apparatus 200 of a second embodiment.

FIG. 17 is a flowchart illustrating processes that the control unit 22informs that the person 500 to be observed is in an abnormal motionstate.

FIG. 18 is a flowchart illustrating processes of determining a situationof a drop or fall of the person 500 to be observed.

FIGS. 19A and 19B are diagrams illustrating a deviation of a pitch anglewhen the camera 11 is installed.

FIG. 20 is a flowchart illustrating processes for correcting a motionvector on the basis of a deviation of the pitch angle of the camera.

FIGS. 21A and 21B are diagrams illustrating the difference between thecamera 11 and a subject and a captured image.

FIG. 22 is a block diagram illustrating the configuration of a detectingsystem of a fifth embodiment.

FIG. 23 is a block diagram illustrating a detailed configuration of theMCU 17 of the fifth embodiment.

FIG. 24 is a block diagram illustrating the configuration of a detectingsystem of a sixth embodiment.

FIG. 25 is a block diagram illustrating a detailed configuration of theMCU 17 of the sixth embodiment.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present invention will be described withreference to the drawings. In the following description, the samereference numerals are designated to the same parts. The names and thefunctions are also the same. Therefore, their detailed description willnot be repeated.

First Embodiment

With reference to the drawings, a detecting system and a detectingapparatus of a first embodiment will be described.

<Configuration>

FIG. 1 is a block diagram illustrating the configuration of a detectingsystem. As illustrated in FIG. 1, a detecting system 100 includes acamera 11, an acceleration sensor 12, a storing unit 13, a notifyingunit 14, a communication module 15, and an MCU 17. For example, in aspace where a person 500 to be observed exists such as a hospital room,the detecting system 100 images the space by the camera 11 and detectsthat the person 500 to be observed makes an abnormal movement on thebasis of the captured image. For example, the detecting system 100detects that the person 500 to be observed drops or falls.

The camera 11 is, for example, a CCD (Charge Coupled Device) imagesensor or CMOS (Complementary Metal-Oxide Semiconductor) image sensorhaving a photodiode, an ADC (Analog to Digital Converter), and the like.The camera 11 sequentially obtains images by imaging and outputs them asvideo data to the MCU 17. For example, the camera 11 is installed in ahospital room, a living room, a bed room, or other space and images theperson 500 to be observed who acts in those spaces.

The acceleration sensor 12 is a sensor for detecting the tilt of theimaging direction of the camera 11 as acceleration in the gravity forcedirection and is, for example, a triaxial acceleration sensor. Theacceleration sensor 12 outputs the detected accelerations in the threeaxes to the MCU 17.

The storing unit 13 is included by a RAM (Random Access Memory) or thelike and, for example, when the MCU 17 performs computation or the likein the detecting system 100, stores data used for the computation.

As illustrated in FIG. 1, an informing unit 16 displays the function ofinforming that the MCU 17 detects the abnormal movement of the person500 to be observed in accordance with the control of the MCU 17. Theinforming unit 16 includes the notifying unit 14 and the communicationmodule 15.

The notifying unit 14 notifies of the information in accordance with thecontrol of the MCU 17. For example, the notifying unit 14 is an LED(Light Emitting Diode), a display, a speaker, or the like and, in thecase where the MCU 17 detects the abnormal movement of the person 500 tobe observed, notifies that the person 500 to be observed drops by avisual method such as a display, an auditory method such as sound outputfrom a speaker, or the like.

The communication module 15 displays a communication function for thedetecting system 100 to communicate with a device on the outside. Forexample, the communication module 15 meets the Bluetooth (registeredtrademark), ZigBee (registered trademark), wireless LAN (Local AreaNetwork) standard, or other wireless or wired communication method andis coupled to an apparatus on the outside by radio or by a wire. In thecase where the MCU 17 detects an abnormal movement of the person 500 tobe observed, the communication module 15 transmits informationindicating that the person 500 to be observed drops to an alarmapparatus 800, a communication apparatus 900, and other externalapparatuses in accordance with the control of the MCU 17. For example,the alarm apparatus 800 receives an alarm signal indicative of theabnormal movement of the person 500 to be observed, which is transmittedfrom the communication module 15 from the detecting system 100 andgenerates an alarm. The communication apparatus 900 receives thenotification indicative of the abnormal movement of the person 500 to beobserved from the detecting system 100 and outputs the information ofthe notification by the display of the communication apparatus 900, thespeaker of the communication apparatus 900, or the like.

FIG. 2 is a block diagram illustrating a detailed configuration of theMCU 17. As illustrated in FIG. 2, the MCU 17 includes a plurality ofinput/output units (I/O units 18A, 18B, 18C, 18D, 18E, and 18F), acapture memory 21, and a control unit (CPU (Central Processing Unit))22. The MCU 17 is coupled to the camera 11, the acceleration sensor 12,the storing unit 13, the notifying unit 14, the communication module 15,a ROM 20, and other external apparatuses and inputs/outputs data from/tothe external apparatuses via the input/output (I/O) units 18A to 18F.Although the storing unit (RAM) 13 and the ROM 20 are illustrated on theoutside of the MCU 17 in FIG. 2, the storing unit (RAM) 13 and the ROM20 may be provided on the inside of the MCU 17.

The capture memory 21 is a memory region for storing video data andstores each of image frames including the video data.

The control unit 22 is a processor for controlling the operation of theMCU 17. For example, the control unit 22 reads and executes a programstored in the ROM 20 on start of the operation of the MCU 17. Afterstart of the MCU 17, the control unit 22 stores the program and datainto the storing unit 13 and operates according to the program.

The MCU 17 receives an entry of video data generated by the camera 11via the I/O unit 18A and holds it in the capture memory 21. The MCU 17obtains the acceleration of the camera 11 from the acceleration sensor12 via the I/O unit 18B and stores the obtained acceleration into thestoring unit 13.

FIG. 3 is a block diagram illustrating functions of a detectingapparatus 200. FIGS. 4A to 4F are diagrams illustrating outlines of thefunctions of the detecting apparatus 200. As illustrated in FIG. 3, inthe detecting system 100 of the first embodiment, the detectingapparatus 200 includes the I/O unit 18, the storing unit 13, the capturememory 21, and the control unit 22.

The capture memory 21 stores video data 41 obtained from the camera 11by the detecting apparatus 200.

The control unit 22 performs a process of detecting the movement stateof the person 500 to be observed on the basis of the video data 41stored in the capture memory 21. The control unit 22 operates inaccordance with the program, thereby displaying the functions of acapture processing unit 31, a thinning processing unit 32, a motionvector calculating unit 33, a flow vector threshold processing unit 34,a labelling unit 35, a characteristic amount calculating unit 36, anabnormal state determining unit 37, and an informing control unit 38.

The capture processing unit 31 performs a process of receiving the videodata generated by the camera 11 by the detecting apparatus 200 andstoring it into the capture memory 21.

Prior to a process of calculating a motion vector (flow vector) of thevideo data 41 which will be described later, the thinning processingunit 32 preliminarily performs a process of reducing a part of pixelsincluding the video data 41. In the first embodiment, the thinningprocessing unit 32 performs, for example, a process of thinning thepixels in the horizontal direction (the direction perpendicular to thegravity force direction) of the video data 41. For example, the thinningprocessing unit 32 performs the thinning by averaging two pixels in thehorizontal direction of the video data 41 to one pixel. In such amanner, the number of pixel blocks as an object of calculation of amotion vector which will be described later is decreased, therebyincreasing the speed of the following processes. As another method, bythinning the pixels in the horizontal direction of the video data 41 bythe thinning processing unit 32, the component in the horizontaldirection of the motion vector is made smaller, and the component of themotion vector in the gravity force direction can be increased. In thecase where the person 500 to be observed drops or falls, the person 500to be observed which is seen as a subject in the video data 41 becomes ablock group having a large motion vector in the gravity force directionin an image. Consequently, by thinning the pixels in the horizontaldirection by the thinning processing unit 32, while maintaining theprecision of detection of a drop or fall of the person 500 to beobserved, a calculation amount of the motion vector can be reduced.

The motion vector calculating unit 33 calculates a motion vector of eachof the plurality of blocks of the image. In the first embodiment, theimage subjected to the thinning process by the thinning processing unit32 is included by the plurality of blocks. Each of the plurality ofblocks includes, for example, eight pixels×eight pixels. However, theinvention is not limited to the configuration. The motion vectorcalculating unit 33 divides an image into blocks of equal intervals andcuts an image block by block. The motion vector calculating unit 33calculates a motion vector by comparing the cut image with, for example,the image of the immediately preceding frame. For example, while moving(shifting) the image which is cut block by block, the cut image iscompared with the image of the immediately preceding frame and aposition (shift position) in which similarity of the images is high isused as the motion vector of the block.

The flow vector threshold processing unit 34 compares the size of eachof the motion vectors for the plurality of blocks and a threshold andselects a motion vector having a predetermined size in the motionvectors. To reduce the influence of a small motion vector caused by achange in brightness or the like of the video data 41 not a drop of theperson 500 to be observed, the detecting apparatus 200 eliminates amotion vector smaller than the threshold from objects to be calculatedby the flow vector threshold processing unit 34 in a process ofdetecting an abnormal movement of the person 500 to be observed.

The labeling (grouping) unit 35 performs grouping (labeling) which formsa group from adjacent blocks of the motion vector having predeterminedsize in the plurality of blocks including the image, and designates aname to the group so that the groups can be identified.

The characteristic amount calculating unit 36 calculates, using motionvectors of the formed block group as an object, characteristic amountsof the blocks such as an average value of gravity-force-directioncomponents of the motion vectors, a variance of the motion vectors, andthe value of a rotation direction component made by the motion vector ofeach block using the center of gravity of the block group as a center.

On the basis of the characteristic amount calculated by thecharacteristic amount calculating unit 36 on each of the group of blocksgrouped, the abnormal state determining unit 37 detects an abnormalmotion state that the person 500 to be observed drops or falls.

In the case where an abnormal motion state of the person 500 to beobserved is detected by the abnormal state determining unit 37, theinforming control unit 38 controls a process of informing that theperson 500 to be observed is in an abnormal motion state such as a dropor fall by the informing unit 16. For example, the informing controlunit 38 informs an external apparatus of the fact that the person 500 tobe observed is in an abnormal motion state by the communication module15.

The functions of the control unit 22 will be concretely described. Asillustrated in FIG. 4A, the capture processing unit 31 stores an imagegenerated by the camera 11 as the video data 41 into the capture memory21. As illustrated in FIG. 4B, the motion vector calculating unit 33calculates a motion vector on each of the blocks of the image includingthe video data 41. In FIG. 4B, each of the motion vectors is expressedby the size and the direction of an arrow. As illustrated in FIG. 4C,the flow vector threshold processing unit 34 selects motion vectorshaving predetermined size or larger (pixel block groups 43A and 43B) andsends them to a grouping process by the labeling unit 35.

As illustrated in FIG. 4D, the labeling unit 35 performs a process ofidentifying adjacent blocks as a group by the group name, having themotion vectors of the predetermined size or larger selected by the flowvector threshold processing unit 34. In the example of FIG. 4D, thegroup name “Group 1” is given to the pixel block group 43A, and thegroup name “Group 2” is given to the pixel block group 43B. Asillustrated in FIG. 4E, the characteristic amount calculating unit 36calculates the characteristic amounts of the motion vectors of eachgroup on the basis of the motion vectors of the groups labeled. In theexample of FIG. 4E, the characteristic amount calculating unit 36calculates, as characteristic amounts of each group, an average vectorand a variance of motion vectors of the blocks included in each groupand the value of the rotation direction component of the motion vectorof each block using the block as the center of gravity of each group asa center.

The abnormal state determining unit 37 compares the characteristicamount of each group calculated by the characteristic amount calculatingunit 36 with the threshold to detect whether the person 500 to beobserved is in an abnormal state of a drop or fall in each group. In theexample of FIG. 4F, the pixel block group 43A is detected that theperson 500 to be observed is in an abnormal state as the person 500 tobe observed drops. In the pixel block group 43B, it is not detected thatthe person 500 to be observed is in the abnormal state.

<Data>

With reference to FIG. 5, data used by the detecting apparatus 200 willbe described. FIG. 5 is a diagram illustrating the data structure ofgrouping information 51 and the data structure of characteristic amountinformation 52.

The grouping information 51 is information indicating the range of ablock group to be grouped by the labeling unit 35. For example, each ofimages including the video data 41 is made of a plurality of blocks in Xrows and Y rows. For example, the block in the row x and the column y asone of the blocks included in the image is expressed as the block (x,y).

Image data 51A is information identifying each of the images of thevideo data 41 included by a plurality of frames. In the example of FIG.5, images are identified by time corresponding to reproduction positionsof the video data.

Group name 51B indicates information identifying each group (name of thegroup) formed by the labeling unit 35 in each image.

A group range 51C indicates blocks included in each group.

The characteristic amount information 52 is a diagram illustratingcharacteristic amounts calculated by the characteristic amountcalculating unit 36 with respect to the block groups formed by thelabeling unit 35.

Image data 52A indicates information for identifying each of images ofthe video data 41 included by a plurality of frames.

Group name 52B indicates information for identifying groups formed bythe labeling unit 35 in each image.

Characteristic amounts 52C indicate characteristic amounts (average ofmotion vectors of blocks including each group, variance of motionvectors, and the value of the rotational direction component of eachmotion vector using the center of gravity of the block group formed as acenter position) calculated by the characteristic amount calculatingunit 36 for each of the groups.

<Operation>

Referring to FIGS. 6 to 14, the operation of the detecting system 100 ofthe first embodiment and the detecting apparatus 200 as a component ofthe detecting system 100 will be described. As illustrated in FIG. 1 andthe like, the detecting system 100 is mounted in a room where the person500 to be observed lives and always images the inside of the room by thecamera 11. For example, the detecting system 100 is mounted in ahospital room or a living room, a bed room, or the like where the person500 to be observed lives a daily life. In the detecting system 100, thedetecting apparatus 200 receives the video data 41 generated by thecamera 11 and detects an abnormal motion state such as a drop of theperson 500 to be observed. When the detecting apparatus 200 detects adrop or the like of the person 500 to be observed, the drop or the likeof the person 500 to be observed is informed by the informing unit 16.For example, the detecting apparatus 200 transmits a signal indicatingthat the person 500 to be observed drops to an external apparatus (suchas the communication apparatus 900 or the like) via the communicationmodule 15. An external apparatus such as the communication apparatus 900receives the notification from the detecting apparatus 200 and informsthat the person 500 to be observed is in an abnormal state by, forexample, display of the display, voice, or the like.

FIG. 6 is a flowchart illustrating the processes of detecting a drop orfall of the person 500 to be observed on the basis of an image capturedby the camera 11 by the control unit 22 of the detecting apparatus 200.

In step S601, the control unit 22 performs an initializing process fordetecting a drop or fall of the person 500 to be observed. For example,the control unit 22 sets an initial value as the value of a counterwhich will be described later.

In step S603, the control unit 22 captures each of images of the videodata output from the camera 11 and stores it into the capture memory 21.

In step S605, the control unit 22 performs a thinning process ofreducing a part of the pixels of an image to be processed. The detailsof the process will be described by using drawings. By performing thethinning process by the control unit 22, the number of pixels as objectsof a motion vector calculating process and the like which will bedescribed later is reduced. In the first embodiment, the control unit 22reduces the pixels in the horizontal direction of each of imagesincluding the video data 41, thereby improving the detection precisionof a motion vector in the gravity force direction. The control unit 22stores the image data obtained by reducing the pixels in the horizontaldirection of each of the images of the video data 41 by the thinningprocess into a memory.

In step S607, the control unit 22 calculates motion vectors of the imagesubjected to the thinning process in step S605 block by block. It isassumed that each of the blocks includes 8 pixels×8 pixels.

In step S609, the control unit 22 compares the size of the motion vectorof each block with the threshold and selects a block having a motionvector larger than the threshold. The threshold is set, for example, onthe basis of the size of a motion vector in the case where the person500 to be observed makes an abnormal motion such as a drop or fall. Bythe process of step S609, the influence of a motion vector which issmaller than the size of the motion vector in the case where the person500 to be observed makes an abnormal motion can be eliminated in theprocess of detecting the state of the motion of the person 500 to beobserved.

In step S611, the control unit 22 performs a process of making eachgroup identifiable by grouping adjacent blocks in the blocks selected instep S609 into one group (block group) and setting the name of eachgroup. The control unit 22 associates the range of the blocks includedin each group with the name of the group and the image and stores themas the grouping information 51 into the memory.

In step S613, the control unit 22 sets priority on the basis of thenumber of blocks (the area of blocks) included in a group for each ofthe groups formed in step S611. Concretely, the control unit 22 sortsthe groups in order of the areas of the groups. The area of each of thegroups is calculated on the basis of the range of the group indicated bythe group information 51. The control unit 22 selects, for example, theupper three groups in the groups sorted in the descending order of theareas.

In step S615, the control unit 22 calculates the characteristic amountof the motion vector on the basis of the motion vectors of the blocksincluded in a group for the groups sorted and selected in step S613. Thecharacteristic amounts of the motion vector of each of the blocksincluded in the group are, for example, the average value of componentsin the gravity force direction of the motion vectors, variance of themotion vectors, and the value of the rotation direction component madeby the motion vector of each block using the block as the center ofgravity of the block group included in the group as a center. Thedetails will be described later with reference to the drawings.

In step S617, the control unit 22 calculates a determination value D fordetermination on the basis of the characteristic amounts of the motionvectors of each group calculated in step S615 and compares thedetermination value D with the threshold to estimate whether or not thesubject shown in each group drops or falls. The determination value D iscalculated by, for example, the following equation (1). In the casewhere the determination value D calculated lies in a range of thethreshold (YES in step S617), the control unit 22 estimates that thesubject indicated by the group drops or falls and advances to theprocess in step S619. In the other case (NO in step S617), the controlunit 22 advances to the process in step S621.

D=αVacc+βVrot−γVvar  Equation (1)

An average value Vacc is an average value of components in the gravityforce direction of the motion vector. A rotation direction componentVrot is the value of the rotation direction component made by the motionvector of each block using the block as the center of gravity of thegroup as a center. The block as the center of gravity of the group ofblocks is the block including the center of gravity when the block groupis expressed in the plane figure.

A variance Vvar is a variance of the motion vectors of the blocksincluded in the group. α, β, and γ are parameters for determining theweight. The average value Vacc is a characteristic amount indicative ofa drop, and the rotation direction component Vrot is a characteristicamount indicative of a drop. Consequently, in the case where the averagevalue Vacc and the rotation direction component Vrot are large, thepossibility that the subject drops or falls is high. In the case wherethe variance Vvar is large, the possibility of erroneous detectioncaused by noise of the video data 41 is high. Therefore, the varianceVvar is set as a suppression term in the calculation formula of thedetermination value D.

In step S619, the control unit 22 increments the value of the counterfor detecting the state of the abnormal motion of the person 500 to beobserved.

In step S621, the control unit 22 determines whether the value of thecounter exceeds a predetermined value or not. The value to be comparedwith the counter value is set so that, for example, in the case where aperiod in which it is estimated that the subject shown in each groupdrops or falls in the process of step S617 continues for a predeterminedperiod (for example, about 0.5 second) or longer, the control unit 22detects that the subject is in an abnormal motion state. In the casewhere video data of one second includes images of 30 frames, thecharacteristic amounts are calculated by the process in step S615 foreach of the images, and the determining process is performed in stepS617, for example, the predetermined value to be compared with thecounter value is set to “15” (corresponding to 0.5 second). In such amanner, in the case where a large motion vector which is not a drop orfall of the person 500 to be observed occurs instantaneously, erroneousdetection can be avoided. In the case where it is determined in stepS621 that the value of the counter exceeds the predetermined value (YESin step S621), the control unit 22 performs the process in step S623. Inthe other case (NO in step S621), the control unit 22 performs theprocess in step S625.

In step S623, the control unit 22 outputs a signal indicating that theperson 500 to be observed is in an abnormal motion state and clears thecount value. For example, the control unit 22 performs a process ofsetting a flag indicating that the person 500 to be observed is in theabnormal motion state. When it is detected that the person 500 to beobserved is in the abnormal motion state, the control unit 22 performs aprocess (informing control) for informing the abnormal state of theperson 500 to be observed by notification of the notifying unit 14,output of the signal to an external apparatus such as the communicationapparatus 900, and another process.

In step S625, the control unit 22 performs a process of clearing thecounter value every period (for example, about 10 seconds) sufficientlylonger than the time in which the person 500 to be observed drops orfalls.

The control unit 22 repeats the processes in the step S603 andsubsequent steps for each of the images including the video data 41.Thinning Process (step S605)

With reference to FIG. 7 and FIGS. 8A to 8D, the details of the thinningprocess described in step S605 will be explained. FIG. 7 is a flowchartillustrating the thinning process of reducing the pixels of each of theimages including the video data 41. The control unit 22 performs thethinning process by calling the thinning process and passing image datato be processed and various parameters such as a reduction rate ofreducing the image data and a direction of scanning the image data. Forexample, the control unit 22 receives information of the tilt of thecamera 11 output from the acceleration sensor 12 and tilts the directionof the scan of the image data from the horizontal direction on the basisof a deviation of the roll angle of the camera 11.

In step S701, the control unit 22 performs a process of initializing thestart position of the scan of the image data, the range of the scan, andthe like. For example, the control unit 22 sets the upper left pixel inpixels included in an image as the start position of the scan.

In step S703, the control unit 22 obtains values of a plurality ofpixels arranged in the scan direction from the scan position on thebasis of designation of the parameter of the reduction rate. Forexample, in the case where the reduction rate is 50% (the values of twopixels are averaged and the average value is set as the value of onepixel), the control unit 22 obtains the value of the pixel in the scanposition and the value of one pixel adjacent in the scan direction.

In step S705, the control unit 22 calculates an average value, anintermediate value, or the like of the values of the pixels obtained instep S703.

In step S707, the control unit 22 stores the result of the calculationin step S705 into a memory.

In step S709, the control unit 22 updates information of a position tobe scanned next on the basis of the scan position and the scandirection. For example, in the case where the reduction rate is 50%, anaverage value is calculated every two pixels. Consequently, the controlunit 22 updates the next pixel in a plurality of pixels as objects ofcalculation of an average value as the position to be scanned next inaccordance with the scan direction.

In step S711, the control unit 22 determines whether all of the pixelsincluded in the scan range have been scanned or not. In the case wherethe process in step S705 has not been completed on all of positions tobe scanned (NO in step S711), the control unit 22 repeatedly performsthe processes in step S705 and subsequent steps. In the other case (YESin step S711), the data of the reduced image subjected to the thinningprocess, which is stored in the memory is returned.

FIGS. 8A to 8D are diagrams illustrating the scan directions in thethinning process. The control unit 22 detects a deviation of the rollangle of the camera 11 on the basis of information of the accelerationof the camera 11 output from the acceleration sensor 12, corrects thedeviation of the roll angle, and performs the thinning process on animage.

FIG. 8A illustrates an example of the video data 41 in the case wherethere is no influence of the deviation of the roll angle of the camera11. FIG. 8B is a diagram illustrating the scan directions in thethinning process in the case where there is no influence of thedeviation of the roll angle of the camera 11. As illustrated in FIG. 8B,in the thinning process, the control unit 22 performs a scan on thepixels included in the image while moving the scan position in thehorizontal direction as illustrated in scan positions 47. When the scanon one line in an image including a plurality of lines is finished, thecontrol unit 22 continues the scan from the head of the following line.

FIG. 8C illustrates an example of the video data 41 in the case wherethe roll angle of the camera 11 is deviated by the angle θ. The imageillustrated in FIG. 8C is influenced by the deviation based on theimaging direction of the camera 11. Consequently, in calculation of thecharacteristic amount of the motion vector in step S615, to calculate anaverage value of the components in the gravity force direction, acorrection has to be made to eliminate the influence of the deviation ofthe roll angle of the camera 11. An example of a correcting method is toturn the video data output from the camera 11 in accordance with thedeviation of the roll angle of the camera 11. Another correcting methodis that the control unit 22 corrects a motion vector calculated by theprocess in step S607 or the like in accordance with the deviation of theroll angle of the camera 11. As further another correcting method, anaverage component in the gravity force direction of a motion vectorincluded in a group, calculated as a characteristic amount in theprocess of step S615 is corrected by the angle θ indicative of thedeviation of the roll angle of the camera 11.

FIG. 8D is a diagram illustrating an example of correcting the scandirection by tilting the scan direction of the image by the control unit22 in the case where the roll angle of the camera 11 is deviated by theangle θ. For example, the control unit 22 sets a range obtained byturning the video data 41 only by the angle θ from the center point ofthe image as a range of executing the scan (scan range 45). Asillustrated in scan positions 48 in FIG. 8D, the scan directions aretilted in correspondence with the deviation (angle θ) of the roll angleof the camera 11.

In the first embodiment, the control unit 22 corrects the deviation ofthe roll angle at the time of imaging of the camera 11 by turning theimage at a stage before the thinning process in step S605 is executed.In the thinning process, by correcting the deviation of the roll angleof the camera 11 and performing the thinning process on the correctedimage, noise other than that in the gravity force direction of the imagecan be minimized.

Calculation of Motion Vector (step S607)

FIGS. 9A to 9C are diagrams illustrating outline of the process ofcalculating the motion vector of each of blocks of an image. Asillustrated in FIGS. 9A to 9C, the motion vector is calculated byreferring to a plurality of frames of an image. FIG. 9A is a diagramillustrating an example of an image including the video data 41. FIG. 9Bis a diagram illustrating an image after one frame of FIG. 9A and a partof blocks included in the image. FIG. 9C is a diagram illustrating amotion vector calculating method.

As illustrated in FIGS. 9A and 9B, it is assumed that a vehicle isincluded as a subjected in preceding and subsequent frames. Asillustrated in FIG. 9A, it is assumed that a block as an object ofcalculating a motion vector is a block 55. As illustrated in FIGS. 9Band 9C, the control unit 22 calculates a motion vector by obtaining howmuch the image included in the block 55 of the present frame is movedfrom the preceding frame. Concretely, an image included in a block ofthe present frame to be processed is cut out, and the cut image (window)is moved in parallel using the cut position as a reference. Each timethe window is moved, the control unit 22 overlaps the image with theimage of the preceding frame and calculates the similarity of theimages. The control unit 22 repeatedly calculates the similarity whilemoving the window and sets a movement position having the highestsimilarity as a motion vector.

Calculation of Characteristic Amounts (step S615)

With reference to FIG. 10 to FIGS. 13A to 13C, the process ofcalculating the characteristic amounts of the motion vector of each ofblocks included in a group in step S615 will be described. On the basisof the motion vectors of the blocks included in the group, the controlunit 22 calculates, as characteristic amounts, the average value of thecomponents in the gravity force direction of the motion vectors,variance of the motion vectors, and the value of the rotation directioncomponent using the block as the gravity center of the blocks includedin the group as a center.

FIG. 10 is a flowchart illustrating the process of calculating anaverage vector of the motion vectors of the blocks included in thegroup. The control unit 22 specifies a block included in the group withreference to the grouping information 51. The control unit 22 performs aprocess of calculating an average vector of motion vectors included inthe group by calling the process for calculating the average vector ofthe motion vectors as a characteristic value and passing a set of themotion vectors included in the blocks as parameters.

In step S1001, the control unit 22 sets a vector acc as the value “0” asthe initializing process.

In step S1003, the control unit 22 sets a pointer “p” designating ablock as the head block in the group as the initializing process.

In step S1005, the control unit 22 reads the motion vector of the blockindicated by the pointer “p” and adds it to the vector acc.

In step S1007, the control unit 22 sets the pointer “p” so as todesignate the next block in the group.

In step S1009, the control unit 22 determines whether reading of themotion vectors of all of the blocks in the group has been completed ornot on the basis of the value of the pointer “p” and the number ofblocks in the group. In the case where there is a motion vector which isnot subjected to addition (NO in step S1009), the control unit 22repeats the processes in step S1005 and subsequent steps, therebyperforming the process of adding the motion vectors in the group. In theother case (YES in step S1009), the control unit 22 performs the processin step S1011.

In step S1011, the control unit 22 calculates the average vector acc bydividing the vector acc by the number of motion vectors in the group(the number of blocks included in the group) and returns the calculatedaverage vector acc.

FIG. 11 is a flowchart illustrating processes of calculating, as acharacteristic amount, variance of motion vectors, on the basis of themotion vectors of blocks included in a group. The control unit 22 callsa process for calculating variance of motion vectors as a characteristicamount and passes, as parameters, a set of the motion vectors includedin the blocks and the average vector acc calculated by the processesillustrated in FIG. 10, thereby performing a process of calculating thevariance of the motion vectors included in the group.

In step S1101, the control unit 22 sets the vector var to the value “0”as an initializing process.

In step S1103, the control unit 22 sets the pointer “p” designating ablock to the head block in the group.

In step S1105, the control unit 22 reads the motion vector of the blockdesignated by the pointer “p” and calculates the difference diff betweenthe read motion vector and the average vector acc.

In step S1107, the control unit 22 adds the square of the differencediff calculated in step S1105 to the vector var.

In step S1109, the control unit 22 sets the pointer “p” so as todesignate the next block in the group.

In step S1111, the control unit 22 determines whether reading of themotion vectors of all of blocks in the group has been completed or noton the basis of the value of the pointer “p” and the number of blocks inthe group. In the case where there is a motion vector which is notsubjected to addition (NO in step S1111), the control unit 22 repeatsthe processes in step S1105 and subsequent steps, thereby performing theprocess of adding the difference diff of the motion vectors in thegroup. In the other case (YES in step S1111), the control unit 22performs the process in step S1113.

In step S1113, the control unit 22 calculates the variance Vvar of thevectors by dividing the vector var by the value “(the number of blocksincluded in the group)−1” and returns the calculated variance Vvar.

FIG. 12 is a flowchart illustrating processes of calculating the valueof a rotation direction component made by a motion vector of each blockon the basis of the motion vectors of blocks included in the group. Thecontrol unit 22 calls a process for calculating the value of therotation direction component of a motion vector in a group as acharacteristic amount and performs a process of calculating the rotationdirection component by passing a set of the motion vectors of the blocksincluded in the group and the block as the center of gravity of thegroup as parameters.

In step S1201, the control unit 22 sets a rotational amount rot1 to thevalue “0” as the initializing process.

In step S1203, the control unit 22 sets the pointer “p” designating ablock to the head block in the group.

In step S1205, the control unit 22 calculates a position vector pos tothe block indicated by the pointer “p” using the block as the center ofgravity as a center.

In step S1207, the control unit 22 calculates a cross product rot2 ofthe motion vector indicated by the pointer “p” and the position vectorpos extending from the block as the center of gravity to the blockindicated by the pointer “p”.

In step S1209, the control unit 22 performs a computation of dividingthe cross product rot2 by the norm of the position vector pos and addsthe computation result to the rotation amount rot1.

In step S1211, the control unit 22 sets the pointer “p” so as todesignate the next block in the group.

In step S1213, the control unit 22 determines whether reading of themotion vectors of all of the blocks in a group has been completed or noton the basis of the value of the pointer “p” and the number of blocks inthe group. In the case where there is a motion vector which is notsubjected to the computation of the rotation amount (NO in step S1213),the control unit 22 repeats the processes in step S1205 and subsequentsteps, thereby performing the process of calculating the rotation amountof each of the blocks. In the other case (YES in step S1213), thecontrol unit 22 performs the process in step S1215.

In step S1215, the control unit 22 calculates the rotation directioncomponent rot1 made by the motion vector of each of the blocks of thegroup by dividing the rotation amount rot1 by the value “(the number ofblocks included in the group)−1” and returns the calculated rotationdirection component rot1.

FIGS. 13A to 13C are diagrams illustrating an example of distribution ofthe motion vectors of blocks included in a group. As illustrated in FIG.13A, in the case where the motion vectors of the blocks included in thepixel block group 43 are aligned in the gravity force direction, thepossibility that the person 500 to be observed drops or falls isrelatively high. Consequently, in the case where the average value ofthe components in the gravity force direction of the motion vectorsincluded in the group is relatively large, the possibility that theperson 500 to be observed drops or falls is high. On the other hand, asillustrated in FIG. 13B, in the case where variance of the motionvectors of the blocks included in the pixel block group 43 is large, thepossibility of noise is high. As illustrated in FIG. 13C, in the casewhere the rotation direction components made by the motion vectors ofthe blocks included in the pixel block group 43 are large, thepossibility that the person 500 to be observed drops or falls is high.In FIG. 13C, a gravity-center block 42 including the center of gravityof the pixel block group 43 is set as the center, and examples of theposition vectors in the blocks are shown as position vectors 44A, 44B,44C, and 44D. On the basis of the motion vectors of the blocks and theposition vectors of the blocks indicated as the position vectors 44A andthe like, the control unit 22 calculates the values of the rotationdirection components of the motion vectors around the gravity-centerblock 42 as a center (in FIG. 13C, the motion vectors of part of theblocks are illustrated).

Informing Control Process

When the control unit 22 outputs a signal indicating that the person 500to be observed is in an abnormal motion state (for example, sets a flagindicative of the abnormal state of the person 500 to be observed) bythe process illustrated in step S623 in FIG. 6, the control unit 22controls a process for informing an abnormal state of the person 500 tobe observed by the informing unit 16.

FIG. 14 is a flowchart illustrating processes of informing the state ofan abnormal motion of the person 500 to be observed by the detectingsystem 100. For example, in the case where the process of step S623 inFIG. 6 is performed, the control unit 22 performs the processillustrated in FIG. 14.

In step S1401, the MCU 17 instructs the notifying unit 14 to notify ofthe abnormal motion state of the person 500 to be observed. Thenotifying unit 14 outputs a signal indicating that the person 500 to beobserved drops by a visual method (for example, display of informationto the display), an auditory method (for example, sound output by aspeaker), or other methods.

In step S1403, the MCU 17 outputs a signal indicating that the person500 to be observed is in the abnormal motion state to the alarmapparatus 800, the communication apparatus 900, and other apparatusesvia the communication module 15. For example, the alarm apparatus 800receives the signal from the communication module 15 and outputs soundof alarm. For example, the communication apparatus 900 is a radiocommunication terminal, receives data transmitted from the detectingsystem 100, and informs of the fact that the person 500 to be observedis in the abnormal motion state by a display, a speaker, or the like.

SUMMARY OF FIRST EMBODIMENT

The ratio of a drop and a fall is relatively high in unexpectedaccidents which occur in the home. Particularly, in the case of elderlypeople, a place where a drop or fall occurs is often a relatively flatplace such as a living room or a hospital room. It is consequentlydesirable to efficiently find that a person to be observed is in a dropor fall state and to enable early handling.

In the detecting system 100 of the first embodiment, in a process ofimaging a person to be observed by the camera, calculating a motionvector from a captured image and, on the basis of the size of the motionvector of the person to be observed, detecting a drop or fall of theperson to be observed, by the functions of the thinning processing unit32, the motion vector calculating unit 33, the flow vector thresholdprocessing unit 34, the labeling unit 35, and the characteristic amountcalculating unit 36, a situation that an abnormal state of the person tobe observed is erroneously detected due to a motion of a backgroundimage, camera noise, a change in brightness, or the like is avoided. Bycalculating the characteristic amounts by the characteristic amountcalculating unit 36 and determining the state of the person 500 to beobserved by the characteristic amount calculating unit 36 on the basisof the characteristic amount, the possibility that the detecting system100 detects a complicated motion accompanying a drop of the person 500to be observed increases.

According to the detecting system 100 of the first embodiment, withoutrequiring the person to be observed to wear or carry a sensor unit orthe like, the state of the person to be observed can be observedcontinuously and, in the case where a drop or fall occurs, it can benotified automatically. That is, by the detecting system 100 of thefirst embodiment, without applying large burden on the person to beobserved, monitoring can be performed always in daily life.

According to the detecting system 100 of the first embodiment, theperson to be observed himself/herself does not have to transmitinformation indicating that he/she is in an abnormal motion state.Consequently, for example, in the case where the person to be observedsuddenly loses consciousness or even in the case where the person to beobserved drops due to an unexpected external force, the information isautomatically detected and notified to an external apparatus and thelike.

According to the detecting system 100 of the first embodiment, a blockis selected by comparing the threshold and the size of a motion vectorin the image including the video data 41. The detecting system 10 groupsblocks and performs calculation of the characteristic amounts of themotion vectors of the blocks of each group as objects. In such a manner,while separating the motion of an object other than the person to beobserved and unnecessary for detection of the state of the person to beobserved (for example, the motion of a curtain in a room), the state ofthe motion of the person to be observed can be detected. Also in anobject whose characteristic of its surface is small such as a whitewall, noise of a motion vector may occur due to a change in brightnessof the camera, shadows, and the like. In the detecting system 100 of thefirst embodiment, the influence of them can be also separated.Consequently, the detecting system can detect an abnormal state of theperson to be observed with high precision. The detecting system 10 setspriority for the blocks grouped on the basis of the areas of the blocksincluded in the group, calculates the characteristic amounts indescending order of the area, and determines the state of the motion ofthe person to be observed. Therefore, in the case where a plurality ofgroups are detected from an image, prior to calculation of acharacteristic amount of a relatively small group which is assumed tohave no relation, a characteristic amount of a group which is assumed tobe the person to be observed can be calculated, and the speed of theprocess of detecting the state can be increased.

The detecting system can realize protection of privacy of the person tobe observed by, for example, not transmitting an image until an abnormalstate of the person to be observed is detected. Consequently, the personto be observed does not have to be aware of observation by a camera indaily life so that psychological burden can be reduced.

In the case where an abnormal state of the person to be observed isdetected, the detecting system can extract the image of a blockdetermined to be in an abnormal state and transmit it to an externalapparatus or the like. In such a manner, privacy of the person to beobserved can be protected.

Second Embodiment

With reference to FIGS. 15 to 18, a detecting system of a secondembodiment will be described. In the detecting system of the secondembodiment, in the case where a plurality of subjects are included in acaptured image and grouped by the labeling unit 35, whether a subject ina group makes an abnormal motion or not is detected, and an alarm levelindicative of the degree of the abnormal state is output. The detectingsystem outputs the alarm level on the basis of the number of groupsincluded in the image, a result of calculation of characteristic amountsof each group, the area of the group, and the like.

FIGS. 15A to 15F are diagrams illustrating an example of the state of adrop or fall of a person to be observed.

FIG. 15A illustrates an example that the person 500 to be observed isalone and drops. For example, the number of groups extracted from animage is one and the number of groups detected as a drop of the person500 to be observed by calculation of the characteristic amounts and thelike is one. In this case, there is the possibility that the person 500to be observed is alone and drops or falls, so that the control unit 22sets the level of alarm to “high” indicating that the alarm level ishigh and sets the class of the state to the alarm class “1”. Further, inthe case where there is no large change in the motion vectors and thereis no motion of the person 500 to be observed after the detection thatthe person 500 to be observed is along and drops or falls, the controlunit 22 may set the alarm level to a higher level.

FIG. 15B illustrates an example that the person 500 to be observed isalone and drops and an image includes a background object such as asubject 510. For example, the number of groups extracted from an imageis plural, the number of groups detected as a drop of the person 500 tobe observed by calculation of characteristic amounts and the like isone, and the other groups have large variance and are not detected as adrop. In this case, in a manner similar to the situation of FIG. 15A,the control unit 22 sets the alarm level to “high” and sets the alarmclass to “2”. In the case where variance of the group is large, thecontrol unit 22 regards that the group having the large varianceindicates a background object or the like, not the person 500 to beobserved, and can ignore it in detection of the state of the motion ofthe person 500 to be observed.

FIG. 15C is a diagram illustrating an example that the person 500 to beobserved is alone and a subject 520 as a background object other thanthe person 500 to be observed drops. The detecting system 100 may detectthat a group is a person or a background object by comparing the area ofeach of groups formed with a threshold larger than the area of a person.In the case where it is detected that a background subject drops in thesituation illustrated in FIG. 15C, the control unit 22 estimates that itis not urgent but is dangerous and the person to be observed needs help,and sets the alarm level “middle” indicating that the alarm level is anintermediate level and the alarm class “3A”. In the case where it isdetected that a group indicating a person drops after the detection thatthe background object falls, or in the case where the group indicating aperson merges with the group of the background object detected to bedropped, the control unit 22 sets the alarm level “high” and the alarmclass “3B”.

FIG. 15D is a diagram illustrating an example that when there is aperson (subject 530) other than the person 500 to be observed and thereare a plurality of persons, a subject belonging to any group drops. Forexample, there are a plurality of groups extracted from an image and thenumber of groups detected as a drop of the person 500 to be observed isone. In this case, the control unit 22 sets the alarm level “low”indicating that the alarm level is low and the alarm class “4A”. In thecase where no motion in each group is detected after detection of a dropof one group (for example, the size of a motion vector of a block groupincluded in the group is smaller than a threshold in an image for apredetermined period (which may be, for example, a few seconds), thecontrol unit 22 estimates that the person to be observed needs help andsets the alarm level “high” and the alarm class “4B”.

FIG. 15E is a diagram illustrating an example that a plurality ofpersons such as the person 500 to be observed and a subject 540 drop atthe same time. For example, it is a case where the number of groupsextracted from an image is plural, and the number of groups detected asa drop of a person is plural. In this case, the control unit 22 sets thealarm level “high” and the alarm class “5”.

FIG. 15F is a diagram illustrating an example that subjects are close toeach other and the case where the control unit 22 detects that thenumber of groups included in the image is one and the area exceeds athreshold, or the case where the number of groups is plural and thedistance between the groups is close. In this case, it is estimated thata subject 550 as a person such as a caretaker exists close to the personto be observed in the same room, and the control unit 22 sets the alarmlevel “low” and the alarm class “6”.

As described above, the detecting system outputs the alarm levelindicative of the level of alarm and the alarm class indicative of theclass of the situation on the basis of the number of groups included inan image, a result of calculation of a characteristic amount of eachgroup, and the like, and holds them in a memory or the like.Consequently, the detecting system can inform the situation of a motionof a person to be observed in accordance with the alarm level and thealarm class with reference to the alarm level and the alarm class heldin the memory or the like. For example, the detecting system can switchsound output from the speaker, display content in the display, or thelike in accordance with the alarm level. The detecting system candisplay the situation of a motion of the person to be observed on thedisplay in accordance with the alarm class.

With reference to FIGS. 16 to 18, the operation of the detecting systemof the second embodiment will be described. FIG. 16 is a flowchartillustrating processes of a drop or fall of the person 500 to beobserved on the basis of an image captured by the camera 11 by thedetecting apparatus 200 of the second embodiment. Since the processesfrom step S601 to step S619 are similar to those described in the firstembodiment, their description will not be repeated. After the controlunit 22 updates the counter value in step S619, the control unit 22performs the process in step S631.

In step S631, the control unit 22 determines whether the value of thecounter exceeds a predetermined value (whether a period in which acharacteristic amount of a group lies in the range of the thresholdexceeds a predetermined frame). In the case where the value of thecounter exceeds the predetermined value (YES in step S631), the controlunit 22 performs the process in step S633. In the other case (NO in stepS631), the control unit 22 performs the process in step S625.

In step S633, the control unit 22 outputs a signal indicating that theperson 500 to be observed is in an abnormal motion state.

In the case where the control unit 22 calculates the characteristicamount of each group and a determination value D is not in the range ofthe threshold in step S617 (NO in step S617), the control unit 22performs the process in step S635.

In step S635, the control unit 22 refers to the alarm class in the casewhere it was informed last time that the state of the motion of theperson 500 to be observed is abnormal. When the situation of the motionof the person to be observed is the case where a plurality of groups aredetected and the number of groups detected as a drop is one (alarm class4A) (YES in step S635), the control unit 22 performs the process in stepS637. In the other case (NO in step S635), the control unit 22 performsthe process in step S625.

In step S637, the control unit 22 switches to the alarm level “high” andthe alarm class “4B” and stores them in the memory or the like. Thesituation corresponds to the case where the control unit 22 detects aplurality of groups, the number of groups detected as a drop is one and,after that, there is no motion in each group, and there is thepossibility that the person to be observed needs help. A counter may beprepared to provide a predetermined period until the alarm class “4A” isswitched to the alarm class “4B”, and the control unit 22 may incrementthe counter value each time the process in step S637 is performed and,when the counter value reaches a predetermined value, switch the alarmclass.

FIG. 17 is a flowchart illustrating processes that the control unit 22informs that the person 500 to be observed is in an abnormal motionstate. The processes illustrated in FIG. 17 are, for example, calledwhen the control unit 22 performs the process in step S633 in FIG. 16 ormay be periodically performed by the control unit 22.

In step S1701, referring to the grouping information 51 and thecharacteristic amount information 52, the control unit 22 determines asituation that the person 500 to be observed drops or falls on the basisof the number of groups, the areas of the groups, and the characteristicamounts of the groups and sets the alarm level and the alarm class inaccordance with the determination result. The details of the processwill be described later with reference to FIG. 18.

In step S1703, the control unit 22 notifies of the state of the motionof the person 500 to be observed in accordance with the alarm level andthe alarm class by the notifying unit 14 or the like.

FIG. 18 is a flowchart illustrating the processes of determining thesituation of a drop or fall of the person 500 to be observed. Theprocesses in FIG. 18 correspond to the process of step S1701 in FIG. 17.

In step S1801, the control unit 22 refers to the characteristic amountinformation 52, calculates the difference between the number of groupsincluded in an image and the number of groups in which variance ofmotion vectors included in the groups exceeds the threshold, and storesthe calculation result as a variable cand indicative of the number ofcandidates of groups including a person. Since there is the possibilitythat a group in which the variance exceeds the threshold is not a personbut noise, the influence of the group detected by noise is eliminatedfrom the total number of groups included in the image.

In step S1803, the control unit 22 determines whether the number ofgroups after the noise is eliminated is one or not. The control unit 22determines whether the variable cand is the value “1” or not, that is,whether the number of groups in which the variance of motion vectorsincluded in the groups is small is one or not. In the case where thevariable cand is the value “1” (YES in step S1803), the control unit 22performs the process of step S1813. In the other case (NO in stepS1803), the control unit 22 performs the process of step S1805.

In step S1805, referring to the grouping information 51, the gravitycenter position is calculated for each of the plurality of groups, andwhether the gravity center positions of the groups are closed to eachother by predetermined distance or less is determined. For example, inthe case where the gravity center positions of the groups is a fewpercent or less or tens percent or less of the total number of blocks inthe horizontal direction in an image, it is determined that the gravitycenter positions of the groups are close to each other by thepredetermined distance or less. A threshold to be compared with theinterval between the gravity center positions of groups is set so that,for example, in the case where persons are positioned close to eachother by the distance of a few meters, the gravity center positions ofthe groups in the image are determined to be close to each other by thepredetermined distance or less. In the case where it is determined thatthe gravity center positions of the groups are close to each other bythe predetermined distance or less (YES in step S1805), the control unit22 performs the process of step S1817. In the other case (NO in stepS1805), the control unit 22 performs the process of step S1807.

In step S1807, the control unit 22 determines that a subject shown in agroup is a person or a background object such as furniture on the basisof the area of the group determined as a drop. Referring to the groupinginformation 51, the control unit 22 obtains the area of the group inwhich the determination value D is determined to be within the thresholdin the process of the step S617 on the basis of the characteristicamounts of the groups and it is determined that the person 500 to beobserved drops or falls. The control unit 22 determines whether the areaof the obtained group exceeds the threshold or not and, in the casewhere it is determined that the area exceeds the threshold (YES in stepS1807), performs the process of step S1823. In the other case (NO instep S1807), the control unit 22 performs the process of step S1809.

In step S1809, the control unit 22 determines whether all of the groupsincluded in the image indicate a drop or not. The control unit 22determines whether the number of groups indicated by the variable candand the number of groups determined as a drop match or not. In the casewhere it is determined that the numbers match (YES in step S1809), thecontrol unit 22 performs the process of step S1811. In the other case(NO in step S1809), the control unit 22 performs the process of stepS1825.

In step S1811, the control unit 22 sets the alarm class “5” and thealarm level “high”.

In step S1813 (in the case of YES in step S1803), the control unit 22determines whether the group indicates a person such as the person 500to be observed or a background object such as furniture. The controlunit 22 calculates the area of a group having small variance withreference to the grouping information 51 and determines whether the areaof the group has a predetermined size as compared with a threshold. Thethreshold to be compared with the area of a group may beset so that aperson and a background object such as furniture whose size is differentfrom a person can be discriminated from each other in an image. In thecase where the area of the group having small variance exceeds thepredetermined size (YES in step S1813), the control unit 22 performs theprocess of step S1819. In the other case (NO in step S1813), the controlunit 22 performs the process of step S1815.

In step S1815, the control unit 22 sets the alarm class “1” or “2” andsets the alarm level “high”. For example, in the case where one group isdetected from an image and is determined that it indicates a drop orfall, the control unit 22 sets the alarm class “1”. In the case where aplurality of groups are detected from an image and one of them isdetermined as a group indicating a drop or fall, the control unit 22sets the alarm class “2”.

In step S1817 (NO in step S1805), the control unit 22 determines whetherthe group determined as a group indicating a drop or fall expresses aperson or a background object such as furniture whose size is differentfrom a person. The control unit 22 obtains the area of the groupdetermined as a group indicating a drop or fall with reference to thegrouping information 51. The control unit 22 determines whether theobtained area exceeds the threshold or not and, in the case where it isdetermined that the area exceeds the threshold (YES in step S1817),performs the process of step S1821. In the other case (NO in stepS1817), the control unit 22 determines the process of step S1819.

In step S1819, the control unit 22 sets the alarm class “6” and thealarm level “low”.

In step S1821, the control unit 22 sets the alarm class “3B” and thealarm level “high”.

In step S1823 (YES in step S1807), the control unit 22 sets the alarmclass “3A” and the alarm level “middle”.

In step S1825 (NO in step S1809), the control unit 22 sets the alarmclass “4A” and the alarm level “low”.

As described above, the detecting system of the second embodiment candetermine the degree of an abnormal state of a person to be observed.The detecting system can notify an observing person such as a caretakeror a family member of an abnormal state of a person to be observed byvisually, aurally, or other various methods in accordance with variousstates of the person to be observed such as a serious state and anon-emergency state. By knowing the degree of the abnormal state of theperson to be observed, the observer can easily determine whetherhandling is immediately necessary or not, and the burden is lessened.The detecting system can also control whether an image is notified ornot in accordance with the degree of the abnormal state of the person tobe observed. In this case, privacy protection can be realized inaccordance with the state of the person to be observed.

Third Embodiment

A detecting system of a third embodiment will be described. Thedetecting system of the third embodiment corrects a deviation in theimaging direction of the camera 11 in accordance with the tilt of thecamera 11.

FIGS. 19A and 19B are diagrams illustrating a deviation of a pitch anglewhen the camera 11 is installed. In FIGS. 19A and 19B, the angle formedby a vector in the imaging axis direction of the camera and a vector inthe gravity force direction is set as an angle θ. The control unit 22receives an output value of the acceleration sensor 12 and, on the basisof the received acceleration of the camera 11, calculates the angle θbetween the vector in the imaging axis direction of the camera and thevector in the gravity force direction.

FIG. 19A is a diagram illustrating the case where the vector in thegravity force direction and the vector in the imaging axis direction ofthe camera form an almost 90 degrees (the angle θ is 90 degrees). Inthis case, when the person 500 to be observed drops, the motion vectoris directed downward (the gravity force direction) of an image capturedby the camera 11. The case where the vector in the gravity forcedirection and the vector in the imaging axis direction of the cameraform an almost 90 degrees as illustrated in FIG. 19A is set as areference. FIG. 19B is a diagram illustrating the case where the vectorin the gravity force direction and the vector in the imaging axisdirection of the camera form an angle θ (θ<90 degrees). When the person500 to be observed drops, the drop of the person 500 to be observed is amotion in the gravity force direction. However, in an image captured bythe camera 11, the drop is a motion in which the deviation between thegravity force direction and the imaging axis direction of the camera isreflected. Consequently, by correcting the motion vector in the imagecaptured by the camera 11 on the basis of the angle θ, the deviation ofthe pitch angle of the camera 11 can be corrected. The control unit 22multiplies the motion vector with a correction factor for correcting thedeviation of the pitch angle in a process of calculating the motionvector illustrated in step S607 in FIG. 6 or the process of calculatinga characteristic amount of the motion vector in step S615.

FIG. 20 is a flowchart illustrating processes for correcting a motionvector on the basis of a deviation of the pitch angle of the camera. Forexample, in the case where the control unit 22 calculates an averagevector of motion vectors of blocks included in a group in step S615 inFIG. 6, using the calculated average vector and the angle θ formed bythe imaging axis direction of the camera 11 and the gravity forcedirection as parameters, the processes illustrated in FIG. 20 areexecuted.

In step S2001, the control unit 22 calculates sin θ on the basis of theparameter of the angle θ formed by the imaging axis direction of thecamera 11 and the gravity force direction.

In step S2003, the control unit 22 multiplies the average vector accwith the value 1/sin θ to thereby calculate a correction vector vect_robtained by correcting the average vector acc and returns the calculatedcorrection vector vect_r.

As described above, in the detecting system of the third embodiment, bycorrecting the motion vector in accordance with the deviation of thepitch angle of the camera, also in the case where the pitch angle of thecamera is deviated, precision of detection of a drop or fall of theperson 500 to be observed can be increased. Consequently, theflexibility of installation of the camera is increased.

Fourth Embodiment

A detecting system of a fourth embodiment will be described. Thedetecting system of the fourth embodiment performs a correction inaccordance with the distance from the camera 11 to a subject.

FIGS. 21A and 21B are diagrams illustrating the distance differencebetween the camera 11 and a subject and a captured image. According tothe length of the distance from the camera 11 to the subject, the sizeof the subject in an image captured by the camera 11 varies. The furtherthe subject from the camera 11 is, the smaller the subject in the imagecaptured by the camera 11 becomes. Consequently, in the case where thesubject is far from the camera 11, a motion vector of the subject tendsto become smaller than that of a subject closer to the camera 11.Therefore, the detecting system of the fourth embodiment corrects thesize of the motion vector in accordance with the distance between thecamera 11 and the subject.

FIG. 21A is a diagram illustrating the relations between the camera 11and the subjects (persons 500A and 500B to be observed). FIG. 21B is adiagram illustrating an example of a captured image. In FIG. 21B, thesize of a subject varies in accordance with the distance between thecamera 11 and the subject. In the case where the installation positionof the camera 11 is fixed, in an image captured by the camera 11, thearea of a group of the persons 500A and 500B to be observed is accordingto the distance between the camera 11 and the person 500A to be observedand the distance between the camera 11 and the person 500B to beobserved. Therefore, the control unit 22 estimates the distance betweenthe camera 11 and each of the subjects (the persons 500A and 500B to beobserved) on the basis of the area of the group. According to the resultof estimation of the distance between the camera 11 and each of thesubjects, for example, the control unit 22 may correct a threshold to becompared with the determination value D calculated on the basis of thecharacteristic amount of the group in step S617. For example, the sizeof a motion vector in a captured image, of a subject which is relativelyfar from the camera 11 is relatively small, so that the threshold to becompared with the determination value D may be decreased.

In the detecting system of the fourth embodiment, by making a correctionaccording to the distance between the camera 11 and the subject, theprecision of detection of a drop or fall of the person 500 to beobserved can be increased.

Fifth Embodiment

A detecting system of a fifth embodiment will be described. Thedetecting system 100 is provided with a microphone. By receiving aninput of sound by the microphone, in the case where a change in theinput level of the microphone is large, it is easier to detect that theperson 500 to be observed is in an abnormal state.

FIG. 22 is a block diagram illustrating the configuration of a detectingsystem of a fifth embodiment. FIG. 23 is a block diagram illustrating adetailed configuration of the MCU 17 of the fifth embodiment.

As illustrated in FIG. 22, the detecting system 100 of the fifthembodiment has a microphone 61. The microphone 61, for example, receivesinput of sound in a room space in which the person 500 to be observedacts and outputs a sound signal to the MCU 17.

As illustrated in FIG. 23, the MCU 17 receives input of the sound signalfrom the microphone 61 via an I/O 18G. The control unit 22 receivesinput of the sound signal from the microphone 61 and calculates a changerate Vacou of the level of the sound signal. The control unit 22calculates the determination value D to be compared with the thresholdin accordance with, for example, the following equation (2) in theprocess of step S615 illustrated in FIG. 6.

D=αVacc+βVrot−γVvar+δVacou  Equation (2)

δ denotes a parameter for determining weight. As described above, in thefifth embodiment, the larger the change of the level of the sound signalis, the more the possibility of detecting that the motion of the person500 to be observed is in an abnormal state is increased.

In the detecting system of the fifth embodiment, in the case where alarge sound is generated by a drop, the drop of the person 500 to beobserved is detected more easily. Also in a situation of a dark place orthe like in which the detection rate of a drop of the person 500 to beobserved based on an image captured by the camera 11 decreases, by usingsound as an input, a drop of the person 500 to be observed can bedetected.

Sixth Embodiment

A detecting system of a sixth embodiment will be described. Thedetecting system 100 has a far-infrared sensor. An output of the sensoris received, and a part of high temperature is set as a condition ofdetecting a drop or fall of the person 500 to be observed.

FIG. 24 is a block diagram illustrating the configuration of a detectingsystem of a sixth embodiment. FIG. 25 is a block diagram illustrating adetailed configuration of the MCU 17 of the sixth embodiment.

As illustrated in FIG. 24, the detecting system 100 of the sixthembodiment has a far-infrared sensor 62. The far-infrared sensor 62, forexample, detects the temperature of the room space in which the person500 to be observed acts and outputs the detection result to the MCU 17.

As illustrated in FIG. 25, the MCU 17 receives a signal from thefar-infrared sensor 62 via an I/O 18H. For example, in a process ofcalculating the average vector of motion vectors of a group shown inFIG. 10, the motion vector of each block may be multiplied with atemperature value to calculate the average vector acc. By the operation,a motion of a non-living material can be excluded from objects ofdetection of a drop or fall of the person 500 to be observed. Also in asituation of a dark place or the like in which the detection rate of adrop of the person 500 to be observed based on an image captured by thecamera 11 decreases, by using an output of the sensor, a drop of theperson 500 to be observed can be detected.

MODIFICATIONS

Although the examples that the detecting system 100 has the accelerationsensor and the far-infrared sensor have been described, the presentinvention is not limited to the examples. The detecting system 100 maybe provided with one or any combination of various sensors such as acolor sensor, a distance sensor, a millimeter-wave sensor, an ultrasoundsensor, and a near-infrared sensor. The detecting system 100 detects adrop or fall of the person to be observed in accordance with an outputresult of those sensors.

As the embodiments have been described as described above, obviously anyof the embodiments may be combined.

Although the invention achieved by the inventors herein has beendescribed concretely on the basis of the embodiments, the presentinvention is not limited to the foregoing embodiments but can bevariously modified without departing from the gist of the presentinvention.

It is to be considered that the embodiments disclosed here areillustrative and not restrictive in all respects. The scope of thepresent invention is defined by the scope of claims rather than theabove description, and all of changes that fall within the meaning andrange of equivalency of the claims are therefore intended to be embracedtherein.

What is claimed is:
 1. A detecting apparatus for detecting a motionstate of a person to be observed from an image captured, comprising: aninput/output unit; a memory for storing video data which is received bythe input/output unit; and a control unit controlling a process ofdetecting a motion state of the person to be observed on the basis ofthe video data stored in the memory, wherein the control unit comprises:a calculating unit calculating a motion vector of each of a plurality ofblocks of an image including the video data; a detecting unit extractinga block group as a set of the blocks in each of which the size of themotion vector exceeds a predetermined value on the basis of thecalculated motion vectors and detecting an abnormal motion state of theperson to be observed on the basis of the motion vectors of the blocksincluded in the extracted block group, and an informing control unitoutputting a signal indicative of an abnormal state of the person to beobserved in the case where the abnormal motion state of the person to beobserved is detected by the detecting unit.
 2. The detecting apparatusaccording to claim 1, wherein the detecting unit comprises: a groupingunit extracting and grouping adjacent blocks in each of which the sizeof the calculated motion vector exceeds a first threshold; acharacteristic amount calculating unit calculating a characteristicamount of a motion vector of each group on the basis of the motionvectors of the blocks including each of the groups formed, and adetermining unit detecting an abnormal motion state of the person to beobserved on the basis of the calculated characteristic amount of themotion vector of each of the groups.
 3. The detecting apparatusaccording to claim 2, wherein the grouping unit sets priority on thebasis of the number of blocks included in one or more groups formed, andthe characteristic amount calculating unit calculates the characteristicamounts of the motion vectors in the groups in accordance with thepriority set for the groups.
 4. The detecting apparatus according toclaim 2, wherein the characteristic amount of the motion vector in eachof the groups calculated by the characteristic amount calculating unitincludes at least any of an average value of gravity force directioncomponents of the motion vectors of the blocks included in each of thegroups, a variance value of the motion vectors of the blocks, and thevalues of the rotation direction components made by the motion vectorsof the blocks including the group in the case where the gravity centerposition of each group is set as a center, and the determining unitdetects the state of the person to be observed on the basis of thecharacteristic amount calculated by the characteristic amountcalculating unit.
 5. The detecting apparatus according to claim 4,wherein the determining unit compares an average value of the motionvectors of the blocks calculated by the characteristic amountcalculating unit with a second threshold and, in the case where theaverage value exceeds the second threshold, detects that the state ofthe person to be observed is an abnormal motion state.
 6. The detectingapparatus according to claim 4, wherein the determining unit compares avariance value of the motion vectors of the blocks calculated by thecharacteristic amount calculating unit with a second threshold and, inthe case where the variance value does not exceed the second threshold,detects that the state of the person to be observed is an abnormalmotion state.
 7. The detecting apparatus according to claim 4, whereinthe determining unit compares a value of the rotation directioncomponent calculated by the characteristic amount calculating unit witha second threshold and, in the case where the value of the rotationdirection component exceeds the second threshold, detects that the stateof the person to be observed is an abnormal motion state.
 8. Thedetecting apparatus according to claim 4, wherein the determining unitdetermines motion states of a plurality of groups and, on the basis of acombination of the determination results of the groups, determines thedegree of an abnormal motion state of the person to be observed.
 9. Thedetecting apparatus according to claim 1, wherein the control unitincludes a thinning processing unit performing a thinning process ofreducing pixels in the horizontal direction, of the video data stored inthe memory, and the calculating unit calculates a motion vector of thevideo data obtained by reducing the pixels in the horizontal directionby the thinning processing unit.
 10. The detecting apparatus accordingto claim 9, wherein the control unit receives information ofacceleration of a camera which generates the video data by theinput/output unit and stores it into the memory, and the thinningprocessing unit corrects at least any of a deviation of a pitch angle ofthe camera and a deviation of a roll angle of the camera on the basis ofthe information of the acceleration for video data stored in the memory,and performs the thinning process on the corrected video data.
 11. Thedetecting apparatus according to claim 1, wherein informing of anabnormal state of the person to be observed by the informing controlunit of the detecting apparatus includes transmission of informationindicating that the person to be observed is in an abnormal state to anexternal communication apparatus.
 12. A detecting system for detecting amotion state of a person to be observed from a video image captured,comprising: a camera generating video data by imaging; a memory forstoring the video data generated by the camera; a control unitcontrolling a process of detecting a motion state of the person to beobserved on the basis of the video data stored in the memory; and aninforming unit informing of an abnormal state of the person to beobserved in the case where the control unit detects an abnormal state ofthe person to be observed, wherein the control unit comprises: acalculating unit calculating a motion vector of each of a plurality ofblocks of an image including the video data; a detecting unit extractinga block group as a set of the blocks in each of which the size of themotion vector exceeds a predetermined value on the basis of thecalculated motion vectors and detecting an abnormal motion state of theperson to be observed on the basis of the motion vectors of the blocksincluded in the extracted block group, and an informing control unitmaking the informing unit perform the informing in the case where theabnormal motion state of the person to be observed is detected by thedetecting unit.
 13. A detecting method of a detecting system fordetecting a motion state of a person to be observed from a video imagecaptured, the detecting system comprising: a camera generating videodata by imaging; a memory for storing the video data generated by thecamera; a processor controlling a process of detecting a motion state ofthe person to be observed on the basis of the video data stored in thememory; and an informing unit, the detecting method comprising the stepsof: calculating a motion vector of each of a plurality of blocks of animage including the video data by the processor; extracting a blockgroup as a set of the blocks in each of which the size of the motionvector exceeds a predetermined value on the basis of the calculatedmotion vectors and detecting an abnormal motion state of the person tobe observed on the basis of the motion vectors of the blocks included inthe extracted block group by the processor, and making the informingunit perform the informing by the processor in the case where theabnormal motion state of the person to be observed is detected by thedetecting step.